{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Machine learning and statistics in astronomy and cosmology\n", "\n", "Tom Charnock\n", "\n", "Institut d'Astrophysique de Paris\n", "

\n", "Slides available at [presentations.charnock.fr/MPP](http://presentations.charnock.fr/MPP)\n", "\n", "
\n", "
\n", " \"Sorbonne\n", "
\n", "
\n", " \"Alliance\n", "
\n", "
\n", " \"IAP\"\n", "
\n", "
\n", " \"CNRS\"\n", "
\n", "
\n", " \"Aquila\"\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Overview\n", "#### Some terminology\n", "#### Statistical modeling\n", "#### Machine learning to accelerate science\n", "- targeting rare objects in photometric surveys\n", "\n", "#### Neural networks in Bayesian analysis\n", "- agnostic inference of the halo mass distribution function\n", "\n", "#### Likelihood-free inference\n", "- neural compression for inference of galatic population models" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## A little terminology" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Neural networks - parameterisable functions \\\\(\\mathscr{f}_{w,a}:x\\in\\mathcal{X}\\mapsto y\\in\\mathcal{Y}\\\\)\n", "\n", "
\n", " \n", " \n", "

\n", " Stacks of non-linear activated weighted sums - \\\\(\\displaystyle n_j^l = \\phi\\left(\\sum_iw_{ji}n^{l-1}_i+b_j\\right)\\\\)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Architectures (and their relation to physics)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Convolutional networks\n", "
\n", "\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Convolutions\n", "
\n", " \n", "
\n", "\n", "##### Translational invariance" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Kernel sizes\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "\n", "##### Causal connections" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Residual connections\n", "\n", "
\n", " \n", "
\n", "\n", "##### Perturbative expansion" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Optimised architectures\n", "\n", "
\n", "
\n", "

U-net

\n", " \n", "
\n", "
\n", "

Inception

\n", " \n", "

\n", "
\n", "
\n", "\n", "##### Scale invariance" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Weight sharing\n", "
\n", " \n", "
\n", "\n", "##### Other invariances (rotational, etc.)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Deep learning\n", "
\n", "
\n", " \n", "
\n", "
Making models describing the distribution of data
\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## A little bit of Bayesian statistics\n", "
\n", "
Distribution of data
\n", "\n", "\\\\[D\\sim P(D)\\\\]\n", "\n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### We want to model \\\\(P(D)\\\\) with some physical parameters, \\\\(w\\\\)\n", "
\n", "
Likelihood of data generated by some parameters
\n", "\\\\[L(D|w,a)\\\\]\n", "
\n", "
Prior belief in possible parameters
\n", "\\\\[p(w|a)\\\\]\n", "\n", "#### \\\\(a\\\\) describes the shape of \\\\(L(D|w,a)\\\\) and \\\\(p(w|a)\\\\)\n", "
\n", "\\\\[p(D, w|a)=L(D|w,a)p(w|a)\\\\]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Bayes' theorem\n", "
\n", "
Probability of particular values of the parameters after obtaining some data
\n", "\n", "\\\\[P(w|D,a) = \\frac{L(D|w,a)p(w|a)}{p(D|a)}\\\\]\n", "\n", "#### How much do we believe that our choice of stochastic model is correct?\n", "
\n", "
Prior belief in possible forms of distributions, \\\\(p(a)\\\\)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### What do we believe is the distribution of data?\n", "\\\\[p(D)=\\int_{E_a}\\int_{E_w}dadw\\,L(D|w,a)p(w|a)p(a)\\\\]\n", "\n", "## Bayesian inference\n", "
\n", "Find the possible models, \\\\(a\\\\), with parameter values \\\\(w\\\\) that make \\\\(P(D)\\approx p(D)\\\\) given samples \\\\(D\\sim P(D)\\\\)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Why do we even care...
we should be talking about neural networks!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
Training a neural network is really trying to fit \\\\(P(x,y)\\\\)
\n", "

\n", "
We're normally interested in modelling the dependence of some part of the data on another part
\n", "\n", "\\\\[P(y|x)=\\frac{P(x,y)}{P(x)}\\\\]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "

\n", "

How can we do that?

" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Relative entropy\n", "\n", "A measurement of the information loss by approximating a distribution, \\\\(P(y|x)\\\\), with some other distribution, \\\\(q(y|x,w,a)\\\\).\n", "\n", "\\\\[\\mathbb{D}[P(y|x)||q(y|x,w,a)]=\\int_{E_y}dyP(y|x)\\ln\\frac{P(y|x)}{q(y|x,w,a)}\\\\]\n", "\n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Identical distributions\n", "\\\\[\\mathbb{D}[P(y|x)||q(y|x,w,a)]=0\\textrm{ means }P(y|x)=q(y|x,w,a)\\\\]\n", "\n", "#### Different distributions\n", "\\\\[\\mathbb{D}[P(y|x)||q(y|x,w,a)]>0\\textrm{ means }P(y|x)\\ne q(y|x,w,a)\\\\]\n", "\n", "#### Non-symmetric (not a real distance metric)\n", "\\\\[\\mathbb{D}[P(y|x)||q(y|x,w,a)]\\ne[\\mathbb{D}[q(y|x,w,a)||P(y|x)]\\\\]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### What is the relative entropy? \n", "\\\\[\\begin{align*}\n", " \\mathbb{D}[P(y|x)||q(y|x,w,a)]&=\\int_{E_y}dyP(y|x)\\ln\\frac{P(y|x)}{q(y|x,w,a)}\\\\\n", " &=-\\int_{E_y}dyP(y|x)\\ln q(y|x,w,a)+\\int_{E_y}dyP(y|x)\\ln P(y|x)\n", " \\end{align*}\\\\]\n", " \n", "#### Entropy of \\\\(P(y|x)\\\\) \n", " \n", "#### Cross entropy between \\\\(P(y|x)\\\\) and \\\\(q(y|x,w,a)\\\\) \n", "The cross-entropy measures the average number of bits of information needed to tell that an event is drawn from \\\\(q(y|x,w,a)\\\\) rather than \\\\(P(y|x)\\\\)\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### We don't actually know \\\\(P(x,y)\\\\) - that's what we want to model\n", "\n", "We just have distribution of observed samples, \\\\(P(\\{x,y\\}_\\textrm{train})\\\\), made up of \n", "\n", "\\\\[\\{x_i,y_i\\sim P(x,y)|i\\in[1,n_\\textrm{train}]\\}\\\\]\n", "\n", "(hopefully very close to \\\\(\\approx P(x,y)\\\\))\n", "\n", "Using this sampling distribution we can rewrite the relative entropy as \n", "\n", "\\\\[\\mathbb{D}[P(y|x)||q(y|x,w,a)]=-\\sum_{i=1}^{n_\\textrm{train}}P(y_i|x_i)\\ln q(y_i|x_i,w,a)+\\sum_{i=1}^{n_\\textrm{train}}P(y_i|x_i)\\ln P(y_i|x_i)\\\\]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "\\\\[\\mathbb{D}[P(y|x)||q(y|x,w,a)]=-\\sum_{i=1}^{n_\\textrm{train}}P(y_i|x_i)\\ln q(y_i|x_i,w,a)+\\sum_{i=1}^{n_\\textrm{train}}P(y_i|x_i)\\ln P(y_i|x_i)\\\\]\n", "\n", "\n", "By minimising the relative entropy we can attempt to bring \\\\(q(y|x,w,a)\\\\) close to \\\\(P(y|x)\\\\)
(or at least \\\\(P(y_\\textrm{train}|x_\\textrm{train})\\\\)).\n", "\n", "Entropy term, \\\\(P(y|x)\\\\) is independent of \\\\(w\\\\), so optimising the parameters of \\\\(q(y|x,w,a)\\\\) is equivalent to minimising the cross-entropy\n", "\n", "\\\\[\\begin{align*}\n", "\\widehat{w}&=\\underset{w\\in E_w}{\\textrm{arg min}}\\,\\mathbb{D}[P(y|x)||q(y|x,w,a)]\\\\\n", "&=\\underset{w\\in E_w}{\\textrm{arg min}}\\,-\\sum_{i=1}^{n_\\textrm{train}}P(y_i|x_i)\\ln q(y_i|x_i,w,a)\n", "\\end{align*}\\\\]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### How can we calculate the cross entropy?\n", "\n", "The likelihood of a training set is\n", "- how often each event, \\\\(y_i\\\\), in the training set occurs (or the probability, \\\\(P(y_\\textrm{train}|x_\\textrm{train})\\\\)) \n", "- the estimation of the probability using the likelihood, \\\\(q(y_i|x_i,w,a)\\\\), for every event\n", "\n", "\\\\[L(y_\\textrm{train}|x_\\textrm{train})=\\prod_{i=1}^{n_\\textrm{train}}q(y_i|x_i,w,a)^{n_\\textrm{train}P(y_i|x_i)}\\\\]\n", "\n", "Taking the logarithm (divided by the number of training samples) gives\n", "\n", "\\\\[\\begin{align*}\n", "\\frac{1}{n_\\textrm{train}}\\ln L(y_\\textrm{train}|x_\\textrm{train})&=\\frac{1}{n_\\textrm{train}}\\ln\\prod_{i=1}^{n_\\textrm{train}}q(y_i|x_i,w,a)^{n_\\textrm{train}P(y_i|x_i)}\\\\\n", "&=\\sum_{i=1}^{n_\\textrm{train}}P(y_i|x_i)q(y_i|x_i,w,a)\n", "\\end{align*}\\\\]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Minimising the cross-entropy is equivalent to maximising the logarithm of the likelihood!\n", "

\n", "

Maximum-likelihood estimation!

\n", "

\n", "\\\\[\\widehat{w}=\\underset{w\\in E_w}{\\textrm{arg min}}-\\ln L(y_\\textrm{train}|x_\\textrm{train})\\\\]\n", "

\n", "Equivalent to minimising the relative entropy and bringing \\\\(q(y|x,w,a)\\\\) as close to \\\\(P(y|x)\\\\) as possible.\n", "


\n", "(This comes with all the pitfalls that are prevalent with MLE)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# What are the outputs of neural networks then?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "\\\\[\\mathcal{f}_{w,a}(x)=\\gamma\\\\]\n", "\n", "where \\\\(\\gamma\\\\) are the parameters of our distribution describing the probability of our data\n", "\n", "\\\\[q(y|x,w,a) \\propto \\textrm{Loss}(\\gamma, y)\\\\]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Mean square error: \\\\(\\gamma\\\\) is the mean of a unit variance Gaussian likelihood for \\\\(y\\\\)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Softmax (sigmoid) output: \\\\(\\gamma\\\\) is the probability of the occurance of \\\\(y\\\\) in a Bernoulli distribution" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Absolute error: \\\\(\\gamma\\\\) is the diversity of a zero centred Laplace distribution for the probability of \\\\(y\\\\)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Are these models any good?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# No..." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Neural networks are highly complex functions with no specific reason to be globally smooth.\n", "\n", "The \\\\(\\gamma\\\\) describing targets \\\\(y\\\\) or \\\\(y'\\\\) from inputs \\\\(x\\\\) and \\\\(x'\\\\) have no reason to be consistent, even if \\\\(x\\\\) and \\\\(x'\\\\) are infinitessimally close." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Very difficult to make a predictive statement about how likely is any \\\\(y\\\\) when given a neural network" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Distribution of spectroscopic redshifts in a spectroscopic catalogue\n", "\n", "
\n", " \"Spectroscopic\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Select patches/pixels from COSMOS2020 by RA and Dec\n", "\n", "
\n", "

HSC $g$, $r$, $i$, $z$, $y$

\n", "

ULTRAVista $y$, $h$, $j$, $ks$

\n", "

CFHT u* 1, 2

\n", "

Error model for pixel values, $\\textsf{pix}$:

\n", "
\n", " $$\\begin{align*}\n", " \\textsf{pix}_i &\\sim P(\\textsf{pix}|\\{\\text{CCD read out}\\}_i)\\\\\n", " & \\approx \\mathcal{N}(\\boldsymbol{\\mu}_{\\textsf{pix},i},\\boldsymbol{\\Sigma}_{\\textsf{pix},i})\n", " \\end{align*}$$\n", "
\n", "
\n", " \"COSMOS2020\n", " \"COSMOS2020 \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", " \"Targeted\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "##### According to data, there should always be more likely to have objects with redshifts between 0 and 1\n", "\n", "
\n", " \"Spectroscopic\n", "
\n", "

The first fits the data better, but we can't see it because the mean and mode are not good measures...

\n", "

If we want a flat prior over physical properties, we need to reweight the effect of the data

\n", "
\n", "
\n", "\n", "
\n", "

The mean of the data distribution
(for a single Gaussian)

\n", "
\n", " \"Unweighted\n", "
\n", "
\n", "\n", "
\n", "

The mean of the reweighted distribution
(for a single Gaussian)

\n", "
\n", " \"Weighted\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# The problem with deep learning for science\n", "\n", "### The size of the model" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### When comparing different models we tend to use statistics such as the Bayesian evidence ratio\n", "\n", "\\\\(\\displaystyle B = (\\ln)\\frac{p(D|a_2)}{p(D|a_1)}\\\\)\n", "\n", "### As physicists we prefer some quantification of simplicity (Occam's razor)\n", "- \\\\(BIC = k\\ln n - 2\\ln L(D|w^*, a)\\\\)\n", "- \\\\(AIC = 2k - 2\\ln L(D|w^*, a)\\\\)\n", "\n", "i.e. we prefer ΛCDM over a polynomial fit with tons of parameters and therefore fits the data perfectly, but is not predictive." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Neural networks will always lose at model comparison stage\n", "\n", "Although predictions can be validated using a test set there is no physical principle which will tell us the general behaviour for any new piece of data... we just have to hope!\n", "\n", "
\n", "
\n", " \n", "

\n", "
\n", "Using any sensible criterion for model selection a neural network with it's loss function will always be rejected on parameter counts and the lack of predictability." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Is there any hope for using machine learning?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Of course there is...\n", "\n", "We can see that neural networks are able to extract ridiculously large amounts of information directly from the data. \n", "\n", "We just need to use the extracted information in a way that does not depend on the neural network as a model or only use them in the case when the model isn't important for scientific results..." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Machine learning accelerated science" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", " \"Targeted\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## How do we choose where to target follow-up observations?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "##### Build a probabilistic model of known spectroscopic objects\n", "##### Condition the model on pixel values from photometric surveys\n", "\n", "##### This is a data model and not a physical model... any physics is correlation and not due to causation" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Modelling the probability of redshifts given the existance of objects\n", "\n", "##### Flexible enough to model the fact that the distribution of objects is skewed\n", "\n", "##### Simple enough that we can perform analytical integrals\n", "\n", "### Mixture of Gaussians\n", "\n", "$$P_\\mathscr{w}(z|\\textsf{obj}_i, \\textsf{pix}_i) = \\sum_k \\alpha_k(\\textsf{pix}_i, \\mathscr{w})\\hskip0.1em\\mathcal{N}\\hskip-0.25em\\left(\\mu_k(\\textsf{pix}_i, \\mathscr{w}), \\sigma_k(\\textsf{pix}_i, \\mathscr{w})\\right)$$ " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Targetting rare objects\n", "\n", "##### Calculate the marginal distribution for the probability of objects given particular properties, i.e.\n", "\n", "### High redshift objects, i.e. $z > 6$\n", "\n", "$$P_\\mathscr{w}(\\textsf{obj}|\\textsf{pix}_i, z>6)=\\int_6^\\infty dz \\,P_\\mathscr{w}(\\textsf{obj}|z, \\textsf{pix}_i)P_\\mathscr{w}(z|\\textsf{pix}_i)$$\n", "\n", "### Naïvely seach for maximal probability of objects\n", "\n", "$$\\hat{i} = \\underset{i\\in\\textrm{Survey}}{\\textrm{argmax}}~P_\\mathscr{w}(\\textsf{obj}|\\textsf{pix}_i, z>6)$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "

Targetting objects by maximising the probability of particular properties over entire field

\n", "\n", "$$\\hat{i} = \\underset{i\\in\\textrm{Survey}}{\\textrm{argmax}}~P_\\mathscr{w}(\\textsf{obj}, z|\\textsf{pix}_i)$$\n", "\n", "
\n", " \"Maximising\n", "
\n", "\n", "

A way to find rare objects

" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "##### The neural network is a model for objects which can be evaluated quickly\n", "\n", "##### It is used to suggest where to obtain new data, but can be cross validated with other methods and complementary data\n", "\n", "##### Any poor follow ups can be added to the model to make predictions better in an active targeting scenario" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Neural networks using in a Bayesian setting" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "(Charnock, Lavaux, Wandelt, Sarma Boruah, Jasche and Hudson 2020)\n", "\n", "
\n", "
\n", " \n", "
\n", "
\n", "
Inferring cosmological parameters and initial conditions of simulations from noisy (horrible) data
\n", "
    \n", "
  • Forward model all understood physics
    • \n", "
  • Make abstract function (NN) made from physically motivated architecture choice
    • \n", "
  • Use high dimensional Markov methods to sample physical properties AND neural network parameters

    • \n", "
\n", "
\n", " Not deep learning anymore - we're just benefitting from the differentiable frameworks and doing old-school* physics 😉\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Our neural physical engine\n", "### A local non-linear biasing of the dark matter\n", "
\n", "\n", "
\n", "$${\\Tiny \\psi_j^{\\ell,m} = A\\left(\\sum_{i=-\\kappa/2}^{i=\\kappa/2}K_{i}^{\\ell,m}\\delta_{j-i}^\\textrm{LPT}+b_j\\right), ~~~ \\ell=0, m=0}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### A functional describing the distribution of halos in voxels\n", "
\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### And a likelihood describing how likely a catalogue could be generated from such a halo mass distribution function\n", "\n", "$$\\begin{align*}\n", " {\\Tiny\\mathcal{L} =}&{\\Tiny \\sum_{j\\in\\textrm{catalogue}}\\log\\left[\\sum_i^N\\frac{\\alpha_{i,j}}{\\sqrt{2\\pi\\sigma_{i,j}^2}}\\exp\\left[-\\frac{\\left(\\log(M_j) - \\mu_{i,j}\\right)^2}{2\\sigma_{i,j}^2}\\right]\\right]}\\\\\n", " &{\\Tiny - V\\sum_{j\\in\\textrm{voxels},i=1}^N\\frac{\\alpha_{i,j}}{2}\\exp\\left[\\frac{\\sigma_{i,j}^2}{2}\\right]\\textrm{erfc}\\left[\\frac{\\log\\left(M_\\textrm{th}\\right) - \\mu_{i,j} - \\sigma_{i,j}^2}{\\sqrt{2\\sigma_{i,j}^2}}\\right].}\n", " \\end{align*}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Use high order Hamiltonian Monte Carlo sampling to infer:" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "##### The cosmological parameters which give rise to a realistic power spectrum\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "##### The initial conditions which could give rise to the distribution of halos we see\n", "

\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "##### The Markov chain of parameter values of the neural network\n", "

\n", "\n", "
\n", "(We can marginalise out these parameters to ignore the effect of the network on the overall physics)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### And therefore the possible values of the halo mass distribution function\n", "

\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Likelihood-free inference\n", "\n", "##### We have seen how powerful neural networks are extracting information \n", "\n", " - we do not really trust their outputs (no way to measure how well they are working)\n", "\n", "##### So let's go back to stats.\n", "\n", " - treat network simply as a summarising function (because they can extract information well)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Approximate Bayesian computation (ABC)\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Density estimate likelihood-free inference (DELFI)\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Information maximising neural networks\n", "(Charnock, Lavaux and Wandelt 2018)\n", "\n", "##### Completely different use for network (not modelling)\n", "\n", "##### Find the function which maximises the information about your parameters\n", "\n", "- Don't actually care what the network outputs, but we can make them nice, i.e. Gaussianly distributed, etc.\n", "\n", "This makes use of the deep learning framework, and the neural network, but for a very different purpose!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Gaussian Fisher information\n", "\n", "\\\\[\\begin{align*}\n", "{\\bf F}_{\\alpha\\beta} &= \\left.\\left\\langle\\frac{\\partial\\ln L(x|y)}{\\partial y_\\alpha}\\frac{\\partial\\ln L(x|y)}{\\partial y_\\beta}\\right\\rangle\\right|_{y=y^\\textrm{fid}}\\\\\n", "&=\\frac{\\partial\\mu(y)}{\\partial y_\\alpha}^T{\\bf C}^{-1}\\frac{\\partial\\mu(y)}{\\partial y_\\beta}\n", "\\end{align*}\\\\]\n", "\n", "##### If we do not know the likelihood \\\\(L(x|y)\\\\), transform the data using a neural network to get the likelihood \\\\(L(z|y,w,a)\\\\) , where \\\\(z=f_{w,a}(x)\\\\)\n", "\n", "Then maximise \\\\(\\ln\\det{\\bf F}_{\\alpha\\beta}\\\\) where\n", "\n", "\\\\[{\\bf F}_{\\alpha\\beta} = \\frac{\\partial\\mu(f_{w,a}(x(y)))}{\\partial y_\\alpha}^T{\\bf C}(f_{w,a}(x(y)))^{-1}\\frac{\\partial\\mu(f_{w,a}(x(y)))}{\\partial y_\\beta}\\\\]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "- Calculate the mean of the derivative of the network outputs with respect to the parameters, \\\\(\\displaystyle \\frac{\\partial\\mu(f_{w,a}(x(y)))}{\\partial y_\\alpha}\\\\)\n", "\n", "- Calculate the covariance of the network outputs with respect to the parameters, \\\\({\\bf C}(f_{w,a}(x(y)))\\\\)\n", "\n", "- We also want to set the scale of the summaries which we can do with a regulariser (which is arbitrary) like\n", "\\\\[\\Lambda_2 = \\|{\\bf C}(f_{w,a}(x(y)))-\\mathbb{I}\\|+\\|{\\bf C}(f_{w,a}(x(y)))-\\mathbb{I}\\|\\\\]\n", "\n", "This is all very easy to do using modern frameworks" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Inferring galaxy morphology from simulations\n", "\n", "##### CFHTLS D1 deep fields\n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Do not know the likelihood for such a field" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Using likelihood-free inference we can obtain inference of model parameters\n", "\n", "##### No need for catalogues\n", "\n", "##### No need for selection cuts" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Just need to make somewhat realistic simulations \n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Train IMNN to extract information from such images about the density of spirals and the density of ellipticals\n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Validate on a realisation of simulated data generated at known model parameters\n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Finally infer model parameters on the real data\n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Conclusions" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "##### Machine learning is great - but its rubbish at doing science\n", "##### It can be, however, be used to enable science:\n", "- to speed up analyses or give indications of where to build models\n", "- to build agnostic data models which can be marginalised out\n", "- for super-informative compression to increase the statistical power of current inference methods" ] } ], "metadata": { "celltoolbar": "Diaporama", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" }, "rise": { "controls": false, "height": "100%", "scroll": true, "width": "100%" } }, "nbformat": 4, "nbformat_minor": 4 }